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METHOD, SYSTEM, AND ARTICLE OF MANUFACTURE FOR 
INTEGRATING STREAMING CONTENT AND A REAL TIME 
INTERACTIVE DYNAMIC USER INTERFACE OVER A NETWORK 

5 Inventors 

Michael Lorenz 
Michael Newman 
Jereme Pitts 
Seth Gilman 

10 Field of the Invention 

The present invention relates, in general, to information systems 
and, in particular, to facilitating a real time dynamic user interface within 
interactive streaming content over a network. 

Background of the Invention 

15 As the developments of networked processing systems, such as 

the Internet, continue to advance more computing power and features 
are available to users. For example, information that historically has 
been limited to TV and Radio broadcasts on home appliances is now 
being made available through on-line computing services. The 

20 combination of networked computers and multimedia interactive graphic 
user interfaces permits the access and display of textual and graphic 
information in a manner, and at a speed, not previously known in the 
industry. 

As a result of these advancements in technology, various 
25 streaming content application solutions now exist to facilitate the 
transfer of streaming content over the Internet, such as Microsoft's 
Windows Media Technologies available from Microsoft Corporation 



Attorney Docket No.: ACRD-01000US0 KJD 
kjd/acrd/1 000/1 000.00 1 



Express Mail No.: EL 498 237 336 US 




2 

located at Redmond, Washington and Real Network's Streaming Media 
Solutions available from Real Networks located at Seattle, Washington. 

While these solutions have made streaming media content 
possible, they are neither designed to, nor sufficient to, support a real 
5 time interactive dynamic user interface. For example in the prior art, 
universal resource locator (URL) commands are used in combination 
with streaming media to change the user interface in a browser. For 
example, a media file containing a song may be downloaded or 
streamed to a user and at the end of media file, a URL command may 

10 be included to direct the browser to a site to purchase the song. These 
systems use static commands or URLs in the encoded content stream 
that need to be indexed prior to delivery to the displayed user interface. 
It is necessary to index streaming media every time a modification is 
made to a non-streaming element of video or audio. As a result, the 

1 5 displayed user interface is neither dynamic nor real time and cannot be 
modified "on the fly" to incorporate new high resolution images or 
DHTML. This requirement of indexing a static command with streaming 
video limits flexibility. For example, if information, such as a price or 
image, is changed often, the media file must be constantly reindexed. 

20 Thus, an e-commerce site may have to constantly index a media file. 

Therefore, there is a desire for a method, system and article of 
manufacture for integrating streaming content and a real time interactive 
dynamic user interface on a processing device network. 

SUMMARY OF THE INVENTION 

25 In an embodiment of the present invention, a method, system and 

article of manufacture synchronizes streaming video, high-resolution 
images, text and DHTML in a user interface. The user interface can 
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detect an embedded code in a media file being downloaded which will 
allow for dynamically changing information, such as images and 
DHTML, without requiring timely re-encoding and reindexing of the 
media file. The displayed user interface may be updated with new 
5 information, such as new high resolution images and/or DHTML, by 
accessing the new information in a database responsive to the 
embedded code. 

According to an embodiment of the present invention, the system 
includes: a) a media server utilizing media streaming technology for 

10 providing video stored in a media file; b) a user interface which may 
utilize DHTML, JavaScript, and VBScript and is displayed by a browser; 
and c) a Common Gateway Interface ("CGI") accessible from a content 
server used to call objects stored in a database. The objects may 
contain rules for transferring information, such as a new high resolution 

15 image or price to the displayed user interface. 

The media file and embedded time code is indexed once, yet the 
displayed user interface can be modified "on the fly" to accommodate 
new DHTML and high resolution images, among other information, 
without requiring reindexing. The images and rules in an object stored 

20 in a database can be changed in real time allowing for a modified 
displayed user interface without requiring reindexing of the media file. 

According to an aspect of the present invention, a method for 
obtaining streaming content from a processing device network is 
provided. The method comprises a step of requesting an interface 

25 program from a first processing device in the processing device 
network. The interface program is downloaded to a second processing 
device in the processing device network. A user interface is displayed 
on the second processing device. The interface program requests a 
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media file from a third processing device in the processing device 
network. The media file, including an embedded code, is then 
downloaded to the second processing device. The interface program 
spawns a process responsive to the embedded code. The embedded 
5 code is parsed into a plurality of code segments by the process. A 
memory location in a data store is queried responsive to the embedded 
code segment. The second processing device responds to the rules in 
the memory location. 

According to an aspect of the present invention, the rules include 
10 updating the displayed user interface with a high resolution image 
stored in the data store. 

According to another embodiment of the present invention, the 
first processing device and the second processing device are different 
processing devices. 
1 5 According to another aspect of the present invention, the second 

processing device is a personal computer having a web browser. 

According to another aspect of the present invention, the second 
processing device is a set-top box coupled to a television. 

According to another aspect of the present invention, the media 
20 file is an advanced streaming format (".ASF") file or a real network 
media (".RN") file. 

According to another embodiment of the present invention, the 
displayed user interface includes a first window, a second window and 
a third window. Video is provided in the first window; a high resolution 
25 image is provided in the second window; and text is provided in the 
third window. 

According to another embodiment of the present invention, the 
third processing device is a media server. 
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According to another embodiment of the present invention, the 
process is a common gateway interface ("CGI") process. 

According to another embodiment of the present invention, the 
embedded code is a metadata time code. 
5 According to another embodiment of the present invention, the 

responding step includes updating the user interface display with a high 
resolution image stored in the data store. 

According to another embodiment of the present invention, a 
system for providing streaming content and a real time interactive 
10 dynamic user interface is provided. The system includes a first 
processing device having a web browser and a data store for storing 
information. A second processing device is coupled to the first 
processing device and the data store. The second processing device 
provides the first processing device with (1) a user interface and (2) a 
15 media file having an embedded code. The user interface detects the 
embedded code during a media file download to the first processing 
device. The second processing device creates a process for retrieving 
the information from the data store which is used to alter a displayed 
user interface. 

20 According to another aspect of the present invention, the first 

and second processing devices are computers. 

According to another aspect of the present invention, the data 
store is a disk drive. 

According to another aspect of the present invention, the 
25 embedded code is a metadata time code. 

According to another aspect of the present invention, the first 
processing device and the second processing device is coupled to the 
Internet. 
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According to another aspect of the present invention, the first 
processing device and the second processing device are coupled to an 
intranet. 

According to still another aspect of the present invention, an 
5 article of manufacture including a computer readable memory is 
provided. The article of manufacture includes a first software program 
for providing content to a client. A second software program provides 
streaming media to a client. A third software program detects an 
embedded code in the streaming media. And a fourth software program 
10 accesses a data store responsive to the embedded code. 

According to still another aspect of the present invention, the 
data store includes a software object having rules and the rules are used 
to update a user interface. 

Other aspects and advantages of the present invention can be 
1 5 seen upon review of the figures, the detailed description, and the claims 
which follow. In the figures, like reference numerals indicate like 
components. 

BRIEF DESCRIPTION OF THE DRAWINGS 

Fig. 1 is a high-level hardware block diagram of an embodiment 
20 of the present invention and illustrates the interaction between the end 
user processing device, content provider processing device, media 
processing device, and data store. 

Fig. 2 is a high level software block diagram of an embodiment 
of the present invention showing the interaction between the end user 
25 processing device, content provider processing device, media 
processing device, and data store. 
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Fig. 3 illustrates a displayed user interface according to an 
embodiment of the present invention. 

Figs. 4-7 are control flow diagrams of a method executable by the 
software and hardware illustrated in Figs. 1 and 2 according to an 
5 embodiment of the present invention. 

DETAILED DESCRIPTION 

Fig. 1 illustrates a high level block diagram of system 90 
according to an embodiment of the present invention. System 90 
includes end user processing device 101, content provider processing 

10 device 102, data store 103, and media processing device 104, which 
are coupled to processing device network 100. 

Processing device network 100 may include both wired and 
wireless connections, either singly or in a combination thereof. In an 
embodiment of the present invention, the processing device network 

15 100 is the Internet (a.k.a. the world wide web). In alternate 
embodiments of the present invention, processing device network 100 
is a cellular network, a television network, or an intranet, singly or in 
a combination thereof. 

Processing devices illustrated in Fig. 1 may include, but are not 

20 limited to, computers, information appliances, cellular telephones, digital 
cameras, printers, copiers, facsimile machines, television set-top boxes, 
personal digital assistants, or equivalents thereof. 

In an exemplary embodiment, processing device 102 is a Dell® 
Dimensions® XPS B series desktop computer (Dell Computer Company, 

25 Round Rock, Texas) having typical software and hardware components 
as known by one of ordinary skill in the art. For example, processing 
device 102 includes a central processing unit, persistent storage, I/O 
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devices including a display and keyboard, a network interface and an 
operating system. 

Software in processing devices of Fig. 1 may be stored in a 
computer readable medium such as read-only memory ("ROM"), random 
5 access memory ("RAM"), virtual memory or other memory technologies, 
singly or in combination thereof. 

In an embodiment, data store 103 is computer readable memory, 
such as a magnetic hard disk drive, floppy disk, optical disk, or other 
read-write data storage technologies, singly or in combination thereof. 

10 Fig. 2 illustrates a high level software block diagram of system 90 

illustrated in Fig. 1 . In an embodiment of the present invention, end 
user processing device 101 stores browser 204 and user interface client 
205a in memory. In an embodiment of the present invention, browser 
204 is Internet Explorer 5.00 available from Microsoft Corporation 

15 located at Redmond, Washington. 

In an embodiment of the present invention, software components 
such as CGI interface 208 and user interface 205 are stored on a 
computer readable medium. 

Content provider processing device 102 includes multiple 

20 software programs. In an embodiment of the present invention, 
Common Gateway Interface ("CGI") software 208 is stored in memory 
of processing device 102. In an embodiment, CGI software 208 includes 
at least one CGI process. CGI software allows scripts or programs to 
run on processing device 102. Typically, CGI scripts are implemented 

25 to control users interactions with a web site. Similarly, processing 
device 102 includes content server 209 and user interface 205. A 
server is a software program that provides service or content to users 
accessing processing device 1 02 through network 1 00. In addition, user 
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interface 205 provides user interface client 205a to processing device 
101. 

Data store 103 as described above is coupled to processing 
device network 100. In an embodiment of the present invention, data 
5 store 103 includes a relational database which is accessed by a 
structured query language ("SQL"). Data store 103 includes a memory 
location 103a. In an embodiment of the present invention, object 210 
is stored at memory location 103a and may include rules which will be 
discussed below. Further, data store 103 may also include high 
10 resolution images digitally represented by Graphical Interface Format 
("GIF") files and/or Joint Photographic Experts Group ("JPEG") files. In 
an alternate embodiment, other multimedia file formats may be stored. 

Media processing device 104 stores streaming video server 200 
and media file 201 in memory. Media file 201 includes an embedded 
15 metadata time code 203. 

In an alternate embodiment of the present invention, streaming 
video server 200 and media file 201 are stored in memory on content 
provider processing device 102. 

In an alternate embodiment, data store 103 is coupled to content 
20 processing device 102. 

When an end user desires to access information available on 
processing device network 100, such as the Internet, the user initiates 
a connection through web browser 204. Web browser 204 then may 
request user interface client 205a from content provider processing 
25 device 102. In an embodiment of the present invention, this may be 
accomplished by entering into browser 204 a universal resource locator 
("URL") address of the web site provided by content server 209. A 
user then may download the user interface client 205a from the web 
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site. The connection from processing device 101 to processing device 
network 100 may be established through an Internet service provider 
("ISP"). The ISP is a service that allows an end user to connect to 
processing device network 100 usually through a modem. 
5 After user interface client 205a is downloaded to processing 

device 101 from processing device 102, a user interface display 300 is 
provided on processing device 101 as illustrated in Fig. 3. User 
interface display 300 is provided by processing device 102 downloading 
content, and in particular, a dynamic hypertext mark up language 

10 ("DHTML") page to processing device 101. Browser 204 reads the 
DHTML page and requests the rest of user interface client 205a from 
content server 209. Content server 209 is a software program which 
provides information, such as a DHTML page, to processing devices 
requesting such information. User interface client 205a in an 

15 embodiment of the present invention includes: a) HTML tags and text; 
b) images in a GIF and/or JPEG file format; c) JAVA script; and d) 
references to objects, such as an address to object 210 in data store 
103. 

Fig. 3 illustrates a user interface display 300 at processing device 
20 101 . User interface display 300 includes windows 301, 302 and 303. 
In addition, user interface display 300 includes controls 307, such as 
play, pause, stop, rewind, forward, volume, mute and display size 
control buttons. In an embodiment of the present invention, window 
301 provides video 304, such as video of a room. Window 302 
25 includes a high resolution image 305, such as an image of a chair for 
sale, and window 303 includes text 306, such as dimensions and price 
of the chair. 
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After user interface display 300 is provided to processing device 
101, user interface client 205a requests media from media processing 
device 104. Media may include audio, video or combination thereof. 
In an embodiment of the present invention, video is provided to window 
5 301 from media file 201 stored in processing device 104. As described 
in detail below, media file 201 is provided by streaming video server 
200. As video 304 is provided to processing device 101 , user interface 
client 205a detects an embedded code, and in particular, a metadata 
time code 203. Generally, the user interface client 205a spawns a CGI 

10 process which is used to access object 210 in data store 103. 
Information in object 210 is then used to update user interface display 
300 in an embodiment of the present invention. For example, object 
210 may include rules to display high resolution image 305 in window 
302 or provide text, such as a price in window 303. 

15 In an embodiment of the present invention, electronic 

representations of high resolution image 305 and text 306 (price) are 
stored as files 305a and 306a in data store 103. In alternate 
embodiments, object 210 may implement other actions. For example, 
object 210 may include code to display a web page or code to be 

20 executed. 

The end user also has the ability to interactively change the 
contents of user interface display 300. The user may click the DHTML 
driven buttons in controls 307. 

Multimedia file 201 in an embodiment of the present invention is 
25 a digitized format of video. In an embodiment, video footage is 
captured using a video work station such as a Media 100, Inc., located 
in Marlbor, Massachusetts, Matrox's Digisuite, located in Borval, 
Canada, or Pinnacle System's DV500, located in Mountain View, 



Attorney Docket No.: ACRD-01 000US0 KJD 
kjd/acrd/1 000/1 000.00 1 



Express Mail No.: EL 498 237 336 US 




12 

California. Media file 201 is a compressed file in an embodiment of the 
present invention. 

The embedded code of media file 201 may be indexed using a 
variety of technologies. For example, if media file 201 is an .ASF file, 
5 Stream Anywhere , available from Sonic Foundry, located in Madison, 
Wisconsin, inserts code 203 at predetermined set intervals in order to 
alter user interface display 300. In an embodiment of the present 
invention, media file 201 may include a plurality of codes 203 at 
predetermined time intervals. The format of metadata time code 203 

10 may include, for example, "Frame4.cgi?WeblD = 1 234567&&frame3". 

The time code includes a number of fields to be parsed and 
processed. For example, "Frame4.cgi" represents an address of a CGI 
process which will process the time code information. The CGI process 
may be located in CGI 208. "?" identifies that the following field is a 

15 variable to be passed to the CGI process. "1234567" is a unique time 
index number used by the CGI process. "&&" identifies that the 
following information is the target address or destination of the 
information obtained by the CGI process. For example, "frame3" 
identifies that the CGI process should display information such as chair 

20 image 305 in window 302 illustrated in Fig. 3. 

User interface client 205, in an embodiment of the present 
invention, uses Active X Com objects to encrypt and access data store 
103. The user interface client 205a and Active X Com objects are 
programmed using Microsoft's Visual Basic according to an embodiment 

25 of the present invention. Some functions used in interface client 205a 
exist in the data store as stored procedures. This was done to gain 
optimum performance of the data for access as known in the art. The 
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complete functionality of the video index GUI is to keep the metadata 
time code commands in a manageable real-time environment. 

Figs. 4-7 are control flow diagrams of a method 350 executed by 
software, hardware, or a user, singly or in combination thereof, 
5 illustrated in Figs. 1 and 2 according to an embodiment of the present 
invention. Figs. 4-7 illustrate a sequence of logic steps represented by 
logic blocks. In alternative embodiments of the present invention, 
additional logic blocks may be included and illustrated logic blocks may 
be omitted. 

10 The method 350 initiates as illustrated by logic block 400. An 

end user requests a user interface client 205 from content server 209 
stored on processing device 102 as illustrated by logic block 401. 
Content server 209 then transfers the user interface client 205a to 
processing device 201 as illustrated by logic block 402. User interface 

1 5 client 205 then requests interface components such as images and code 
to display initial user interface as illustrated by logic block 403a. User 
interface display is then displayed as illustrated by logic block 404. For 
example, user interface display 300 as illustrated in Fig. 3 is provided 
to end user processing device 101. User interface client 205a then 

20 requests media from streaming media server 200 on media processing 
device 104 as illustrated by logic block 405. Streaming video server 
200 then begins to provide video to end user interface client 205a 
stored in end user processing device 101 as illustrated by logic block 
406. In an embodiment of the present invention, end user interface 

25 client 205a buffers streaming media for a few seconds as illustrated by 
logic block 407. Video such as video 304 in Fig. 3 is then provided in 
window 301 as illustrated in Fig. 3 and described in logic block 408. 
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A unique embedded code in media file 201 or metadata time code 
203 is then provided by streaming video server 200 and media 
processing device 104 and as illustrated in logic block 501 of Fig. 5. 
User interface client 205a then receives the code 203 as illustrated by 
5 logic block 502. Interface client 205a then determines whether code 
203 is a valid command as illustrated by logic block 503. If the code 
203 is not a valid command, logic transitions back to logic block 501. 
If the command is valid, interface client 205a spawns a process to 
handle the command in code 203 as illustrated by logic block 504. In 

10 an embodiment of the present invention, interface client 205a spawns 
a CGI process. The CGI process parses the metadata time code 
command 203 as illustrated by logic block 505. A segment of the code 
203 is then used to perform a query on data store 103. In an 
embodiment of the present invention, a segment of code 203 is used in 

15 an SQL query to a relational database in data store 103. In an 
embodiment of the present invention, the query determines the contents 
of object 210 in data store 103. Object 210 may include rules for 
updating user interface display 300. For example, object 210 may 
include rules for displaying image 305 and text 306. 

20 Data store 103 receives a command from CGI process as 

illustrated in logic block 604. The commands could include an SQL 
query or update command. A determination is then made by the data 
store whether or not the command is recognized as illustrated in logic 
block 606. If the data store recognizes the command, control transitions 

25 to logic block 607 where the data store searches for the results based 
on the command. Otherwise, logic transitions to logic block 61 0 where 
the data store returns a negative result and control then passes to logic 
block 61 1. If control transitions from logic block 607, a determination 
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is made whether data search is positive as illustrated by logic block 
608. If the data store search is not positive, control transitions to logic 
block 610. Otherwise, control transitions to logic block 609. The data 
store returns a positive search result as illustrated by logic block 609. 
5 These results could include software code or images. The result of the 
command is then returned to the CGI process as illustrated by logic 
block 61 1. A determination is made whether the CGI process updates 
the interface in logic block 507. Logic block 509 represents the CGI 
process action which is used to update user display interface 300. If 

1 0 the CGI process does not change user interface display, logic transitions 
to logic block 508. 

User interface client 205a then receives the command from CGI 
process to update user interface display 300 as illustrated by logic block 
600 in Fig. 6. User interface client 205a then requests needed 

1 5 components, such as an image, to update user interface display 300 
from content server 209 as illustrated by logic block 601. Content 
server 209 then sends user interface client 205 the needed 
components, such as images or code, as illustrated by logic block 602. 
The end user interface display 300 is then updated as illustrated by 

20 logic block 603. 

Fig. 7 illustrates a control flow diagram for the operation of 
control functions 307 illustrated in Fig. 3. An end user activates an 
interface function such as a play button in control buttons 307 as 
illustrated by logic block 700. Interface client 205a then checks if the 

25 function is valid as illustrated by logic block 701 . If the function is not 
valid, the logic transitions to the end 705. Otherwise, the user interface 
client 205a calls the function as illustrated in logic block 702. These 
functions include: playing the video, pausing the video, stopping the 
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video, rewinding the video, fast forwarding the video, increasing 
volume, decreasing volume, muting volume and increasing display size. 
The function then executes as illustrated by logic block 703. Interface 
then reflects the function execution as illustrated in logic block 704 and 
5 control ends as illustrated by logic block 705. 

While certain preferred embodiments of the invention have been 
described, these embodiments have been presented by way of example 
only, and are not intended to limit the scope of the present invention. 
Although the present invention solves the problem of providing 

10 a real time interactive dynamic user interface with streaming content, 
its possible implementations are not limited to this. Using similar 
systems and methods described herein, the streaming content unique 
time code can be easily integrated into various other solutions, e.g. e- 
commerce systems, enterprise resource planning systems, and any 

1 5 other real time database driven systems. 

The foregoing description of the preferred embodiments of the 
present invention has been provided for the purposes of illustration and 
description. It is not intended to be exhaustive or to limit the invention 
to the precise forms disclosed. Obviously, many modifications and 

20 variations will be apparent to practitioners skilled in the art. The 
embodiments were chosen and described in order to best explain the 
principles of the invention and its practical applications, thereby 
enabling others skilled in the art to understand the invention for various 
embodiments and with the various modifications as are suited to the 

25 particular use contemplated. It is intended that the scope of the 
invention be defined by the following claims and their equivalents. 
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